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Acquisition  reform  and  the  implementation  of  agile  acquisition  processes 
within  the  Air  Force  are  allowing  acquisition  professionals  greater  flexibility 
in  meeting  user  requirements.  A strong  emphasis  is  being  placed  on  using 
mature  systems  and  technologies,  allowing  new  programs  to  be  initiated  at 
any  point  in  the  acquisition  process  continuum.  Changes  have  been  incor- 
porated into  the  test  and  evaluation  (T&E)  process  to  support  the  increased 
flexibility.  Judicious  use  of  the  various  types  of  recognized  tests  allow  the 
program  manager  (PM)  to  reduce  risk  and  ensure  performance  expecta- 
tions are  met.  The  following  provides  an  overview  of  the  test  tools  available 
to  the  acquisition  professional  and  highlights  the  evolutionary  changes  that 
were  recently  incorporated  into  Air  Force  test  guidance. 


Headquarters  Air  Force/Test  and  Evaluation  (HQ  AF/TE)  released  a new  test 
and  evaluation  instruction  (Air  Force  Instruction  [AFI]  99-103)  on  August  6, 
2004.  The  release  of  AFI  99-103  corresponded  with  a rewrite  of  AFI  63-101, 
Capabilities  Based  Acquisition,  April  1,  2004  (interim  approval)  and  AFI  10-601, 
Capabilities  Based  Requirements  Development,  July  30,  2004,  bringing  all  the  instructions 
in  line  with  direction  found  in  Chairman  Joint  Chiefs  of  Staff  Instructions  (CJCSIs) 
3170.01D  (2003),  Joint  Capabilities  Integration  and  Development  System,  and  6212.01C, 
Interoperability  and  Supportability  of  Information  Technology  and  Nation  Security  Sys- 
tems. Perhaps  more  importantly,  the  new  AFI  99-103  (2004)  incorporated  evolutionary 
changes,  reflecting  the  test  and  evaluation  community’s  adaptations  to  the  Air  Force’s 
implementation  of  the  agile  acquisition  processes  over  the  last  decade.  The  instruc- 
tion also  poses  a seamless  verification  process  that  fosters  an  integrated  testing 
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philosophy  in  an  effort  to  streamline  Test  and  Evaluation  (T&E),  much  as  the  ac- 
quisition process  itself  is  being  streamlined. 

This  effort  to  integrate  testing  is  born  out  by  the  fact  that  the  new  AFI  99-103 
(2004)  is  itself  a compilation  of  the  former  AFI  99-101  (1996),  Developmental  Test 
and  Evaluation;  AFI  99-102  (1998),  Operational  Test  and  Evaluation;  and  AFI  99- 
105  (1994),  Live  Fire  Test  and  Evaluation.  The  following  shows  how  the  traditional 
T&E  process  supports  the  spiral  acquisition  philosophy,  describes  how  the  types  of 
tests  support  the  new  acquisition  philosophy,  and  highlights  new  test  programs  that 
facilitate  technology  transition  to  the  warfighter. 


TRADITIONAL  TEST  AND  EVALUATION  SUPPORT 
FOR  SPIRAL  ACQUISITION 

Though  the  acquisition  process  has  evolved  along  with  the  country  itself,  the  structure 
for  much  of  the  modem  acquisition  processes  was  put  in  place  by  the  McNamara  re- 
forms and  remains  fundamentally  unchanged.  This  was  done  to  reign  in  a system  that 
was  perceived  as  out-of-control,  as  evidenced  by  aircraft  cost  overruns  that  were  as  much 
as  100  percent  after  inflation  adjustment  toward  the  end  of  the  Korean  War  (Harman, 
1997).  Standardized  procedures  and  processes  replaced  those  developed  by  services  and 
individual  program  offices.  Programs,  especially  major  systems  procurements,  were 
expected  to  adhere  rigorously  to  this  process  (shown  in  Figure  1).  To  progress  through 
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the  process,  a program  had  to  pass  through  a series  of  well-defined  gates,  designed 
to  minimize  risk  to  the  government.  The  program  had  to  demonstrate  its  maturity  by 
passing  a series  of  tests  at  each  gate  that  were  designed  to  ensure  the  development  was 
progressing  adequately.  The  tests  were  on  a continuum  that  focused  initially  on  the 
engineering  aspects  of  the  program  and  shifted  later  to  operational  concerns. 

Requirements  under  United  States  Code  Title  10  §2399  (2002)  drew  a clear  distinc- 
tion between  the  two  types  of  testing  by  requiring  the  user  to  demonstrate  the  effec- 
tiveness of  the  final  design  in  an  operational  environment.  This  further  dichotomized 
developmental  test  and  evaluation  (DT&E),  where  the  acquisition  community  tested 
systems  and  subsystems  against  engineering  and  contract  specifications,  and  opera- 
tional test  and  evaluation  (OT&E),  where  operators  tested  the  entire  systems  including 
its  support  infrastructure  against  the  mission  requirements.  This  dichotomy  was  em- 
phasized by  the  requirement  for  the  DT&E  test  organization  to  certify  the  system  as 
ready  for  operational  testing  (see  AFMAN  63-119). 


The  DT&E  portion  of  the  test  and  evaluation 
process  has  been  inherently  more  flexible  by 
virtue  of  having  the  requirements  developed 
internally  to  the  program  offite  and  tan 
torrespondingly  adapt  to  the  requirements 
of  a spiral  atquisition  professes. 


The  DT&E  portion  of  the  test  and  evaluation  process  has  been  inherently 
more  flexible  by  virtue  of  having  the  requirements  developed  internally  to  the 
program  office  and  can  correspondingly  adapt  to  the  requirements  of  a spiral 
acquisition  processes.  The  program  office’s  design  integrated  product  team  (IPT) 
is  delivered  a set  of  mission  requirements  for  the  program,  along  with  any  as- 
sociated key  performance  parameters  (KPPs).  The  design  IPT  takes  the  mission- 
level  requirements  and  develops  system  design  requirements  and  specifications 
that  are  traceable  to  the  KPPs.  The  program  office  now  has  the  latitude  to  select 
which  design  requirements  and  specifications  are  critical  technical  parameters 
(CTPs),  subject  to  the  milestone  decision  authority  (MDA)  approval  (and  Office 
of  the  Secretary  of  Defense  [OSD]  approval  for  programs  on  the  oversight  list). 

An  example  of  a KPP  and  its  derived  CTPs  could  be  the  user  requirement  for 
supersonic  cruise  capability  in  military  power  and  the  respective  engineering 
requirements  for  a maximum  take-off  weight  and  minimum  thrust  to  accomplish 
this  cruise  capability.  The  CTPs  normally  drive  the  DT&E  test  requirements  and 
the  program  office  gets  to  decide  what  level  and  degree  of  testing  is  required. 
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If  the  contractor  test  is  deemed  adequate,  data/report  review,  over-the-shoulder 
observation,  or  test  participation  may  be  the  extent  of  government  involvement.  If 
government  testing  is  required,  a DT&E  agency  will  be  given  the  requirements  to 
execute  at  decision  points  within  the  development  effort.  Since  the  program  office 
owns  the  design  process,  chooses  which  parameters  to  track/test,  and  controls  what 
goes  to  the  DT&E  process,  their  trade  space  in  making  those  decisions  includes 
impacts  due  to  the  DT&E  test  requirements. 

The  two  biggest  changes  to  the  DT&E  process  stem  from  the  integrated  test  approach 
espoused  in  AFI  99-103  (2004)  and  the  preference  to  employ  combined  DT&E  and 
OT&E,  whenever  practical.  Collaboration  between  the  designated  OT&E  and  DT&E 
agencies  has  been  historically  focused  on  the  development  of  the  test  and  evaluation 
master  plan  (TEMP)  and  preparation  for  certification  for  dedicated  OT&E.  The  new 
requirement  to  establish  an  integrated  test  team  (ITT)  at  the  onset  of  a program  will 
ensure  contractor,  developmental,  and  operational  testers  remaining  actively  en- 
gaged throughout  the  program.  This  will  assist  in  obtaining  the  goal  of  seamless 
verification  by  helping  to  eliminate  duplicate  test  requirements,  correcting 
scheduling  inefficiencies,  identifying  performance  concerns  early,  and  smooth- 
ing the  transition  to  dedicated  OT&E.  An  ITT  will  also  facilitate  the  accomplish- 
ment of  combined  DT&E  and  OT&E  by  ensuring  OT&E  requirements  are  commu- 
nicated early  and  incorporated  into  DT&E  planning. 


Numerous  failings  of  t ritual  operation  issues 
(COIs)  during  operational  testing  have  resulted  in 
detertifying  the  systems  and  sending  them  batk 
into  development  to  (orrett  the  defidendes. 


OT&E  test  agencies  have  traditionally  become  heavily  involved  in  active  testing 
following  certification  for  dedicated  OT&E.  Numerous  failings  of  critical  operation 
issues  (COIs)  during  operational  testing  have  resulted  in  decertifying  the  systems 
and  sending  them  back  into  development  to  correct  the  deficiencies.  This  has  prompted 
the  Director,  OT&E  (OSD)  and  HQ  AF/TE  to  push  for  more  active  participation  by 
OT&E  agencies  earlier  in  the  development  process.  This  has  led  to  the  implemen- 
tation of  numerous  types  of  user  demonstrations  that  are  executed  in  parallel  with 
the  development  process,  such  as  Early  Operational  Assessments  (EOAs),  Opera- 
tional Assessments  (OAs),  and  Operational  Utility  Evaluations  (OUEs),  which  are  all 
authorized  under  AFI  99-103.  Operational  testers  have  also  been  encouraged  to 
participate  in  DT&E  tests  as  observers  to  facilitate  early  communication.  (Note:  The 
legal  requirements  for  operational  testing  are  to  support  production  and  fielding 
decisions  at  the  end  of  the  acquisition  process.) 
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OT&E  has  begun  to  have  a profound  impact  on  DT&E  by  program  offices 
implementing  combined  DT&E/OT&E  for  cost  and  schedule  efficiencies.  Policies 
implemented  as  the  result  of  the  legal  requirement  for  operational  testing  preclude 
contractor  participation  in  the  execution  of  the  test  (except  as  planned  for  in  the 
concept  of  operations),  data  collection  and  analysis  by  the  prime  contractor,  the  use 
of  prototype  hardware,  use  of  a non-representative  test  environment,  and  changes 
to  the  test  articles.  None  of  these  prohibitions  apply  to  DT&E,  but  must  be  observed 
during  combined  testing  to  carry  data  forward  into  OT&E  analysis. 


There  are  two  types  of  qualifitation  testing: 
qualifitation  test  and  evaluation  (QT&E), 
analogous  to  DT&E,  and  qualifitation 
operational  test  and  evaluation  (QOT&E), 
analogous  to  initial  OT&E, 


Air  Force  99-series  instructions  and  regulations  have  historically  allowed  for  the 
performance  of  qualification  testing.  Qualification  testing  was  designed  to  facilitate 
procurement  of  non-developmental  items  (NDI),  ready  to  be  put  to  use  in  the  Air 
Force.  There  are  two  types  of  qualification  testing:  qualification  test  and  evaluation 
(QT&E),  analogous  to  DT&E,  and  qualification  operational  test  and  evaluation 
(QOT&E),  analogous  to  initial  OT&E.  Successful  completion  of  a QOT&E  provided 
results  for  milestone  C entry  for  a non-developmental  item.  The  availability  of  QT&E 
allowed  for  earlier  testing  of  an  NDI  system  that  may  require  minor  modification  or 
perhaps  had  some  issue  concerning  readiness  for  QOT&E.  The  existence  of  quali- 
fication testing  demonstrates  flexibility  in  the  existing  process  to  support  some  of  the 
currently  touted  agile  acquisition  philosophies.  It  just  lacked  the  present  emphasis. 


EVOLUTIONARY  TEST  TOOLS 

One  of  the  primary  goals  of  agile  acquisition  is  to  shorten  the  acquisition  cycle 
time.  This  can  be  accomplished  by  either  reducing  the  time  it  takes  to  accomplish 
the  process  or  by  entering  the  acquisition  cycle  at  a later  stage  in  the  process.  To 
accomplish  the  later,  OSD  has  implemented  a hierarchy  of  solutions  for  meeting 
requirements  via  a new  start  acquisition  program.  This  directs  decision  makers  to 
look  for  solutions  where  development  costs  are  minimal  and  design  solutions  are 
mature.  This  hierarchy  is  listed  below  in  order  of  descending  preference: 
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1.  Use  or  modification  of  an  existing  U.S.  system, 

2.  Use  or  modification  of  an  existing  commercial  or  allied  system, 

3.  Cooperative  development  with  an  allied  nation, 

4.  Joint  service  development  program,  and 

5.  Service-unique  development  program. 

Though  the  preference  for  use  of  existing  technologies  is  clear,  there  is  no  direct 
incentive  for  their  selection.  To  provide  the  incentive  to  use  such  solutions,  both 
OSD  and  the  Air  Force  have  created  special  test  programs  and  processes  that  provide 
both  funding  and  empowerment.  This  encourages  program  offices  to  select  solutions 
capable  of  entering  the  acquisition  process  at  milestone  B or  C with  a demonstrated 
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system  capability.  The  OSD  pregrams  include  the  Foreign  Comparative  Test  (FCT) 
and  Advanced  Concept  Technology  Demonstration  (ACTD).  The  Air  Force  created 
mission-oriented  battlelabs. 

The  FCT  Program  is  managed  by  the  Comparative  Testing  Office  (CTO)  under 
the  Deputy  Under  Secretary  of  Defense,  Advanced  Systems  & Concepts,  Office  of 
the  Under  Secretary  of  Defense  (Acquisition,  Technology  & Logistics).  The  FCT 
program  brings  value  to  the  acquisition  process  by  using  foreign  military  and  com- 
mercial non-developmental  items  to  enter  the  process  at  milestone  B or  C and  bypass 
all  or  part  of  the  development  process.  The  FCT  Program  is  authorized  by  Title  10, 
United  States  Code,  Section  2350a(g)  and  funded  by  Office  of  the  Secretary  of 
Defense  (OSD)  Research,  Development,  Test  and  Evaluation  (RDT&E)  appropria- 
tions. The  objectives  of  the  FCT  program,  as  stated  in  the  handbook,  are  to  improve 
U.S.  warfighter’s  capabilities  and  reduce  expenditures.  The  handbook  provides  cradle 
to  grave  instruction  for  an  FCT  and  can  be  found  at  the  CTO’s  Web  site  http:// 
www.acq.osd.mil/cto. 


The  purpose  of  an  ACTD  is  to  demonstrate  the 
effe<tiveness  of  mature  or  maturing  tethnology 
to  meet  a (ritual  user  need . 


Two  categories  of  FCTs  are  authorized:  procurement  testing  and  technical 
assessment.  Procurement  testing  focuses  on  finding  a materiel  solution  for  an 
existing  requirement.  Although  technical  assessments  for  the  examination  of  po- 
tentially revolutionary  foreign  technologies  are  allowed,  priority  is  given  to  projects 
that  would  result  in  the  initiation  of  an  acquisition.  There  are  also  two  types  of 
procurement  testing:  qualification  and  comparative  testing.  Qualification  testing 
determines  if  a prospective  system  meets  the  stated  mission  requirements;  while 
comparative  testing  performs  side-by-side  testing  on  multiple  items  that  are  poten- 
tially capable  of  meeting  the  requirement.  If  domestic  items  are  potential  candi- 
dates in  the  comparison  test,  the  program  sponsor  must  ensure  funds  are  available 
to  execute  that  portion  of  the  test  before  final  approval  of  the  program  as  an  FCT. 
Numerous  examples  of  programs  that  have  completed  the  FCT  process  are  found 
on  the  CTO  Web  site.  FCTs  have  led  to  more  than  $6.2  billion  in  procurements, 
saving  an  estimated  $4.4  billion  in  development  costs. 

Advance  Concept  Technology  Demonstration  (ACTD)  program  is  another  OSD 
program  managed  by  the  Deputy  Under  Secretary  of  Defense  for  Advanced  Sys- 
tems and  Concepts  (DUSD(AS&C)).  The  purpose  of  an  ACTD  is  to  demonstrate 
the  effectiveness  of  mature  or  maturing  technology  to  meet  a critical  user  need. 
The  potential  effectiveness,  maturity  of  their  respective  technologies,  and  the  user 
needs  being  met  are  key  criteria  against  which  candidate  programs  are  evaluated. 
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Candidate  technologies  with  a high  degree  of  maturity  and  that  could  have  revolution- 
ary impacts  on  operational  capabilities  have  an  excellent  chance  of  being  selected  and 
funded  under  this  program. 

Each  ACTD  is  executed  by  a lead  agency  and  must  be  sponsored  by  a user.  The 
Joint  Requirements  Oversight  Council  (JROC)  will  review  the  candidate  ACTDs  and 
make  recommendations  concerning  their  selection  and  execution.  Programs  selected 
will  nominally  receive  10  to  30  percent  of  the  ACTD’s  funding  from  OSD.  Each  ACTD 
will  be  executed  under  an  oversight  group,  chaired  by  DUSD(AS&C).  A major  ob- 
jective of  the  ACTD  program  is  to  transition  technologies  with  demonstrated  military 
utility  seamlessly  into  acquisition  programs.  With  the  requirement  for  the  technology 
to  be  mature,  in  order  to  qualify  for  an  ACTD,  the  program  should  enter  the  traditional 
acquisition  cycle  late  in  the  continuum  and  reduce  the  time  to  field  the  system.  However, 
since  the  technology  is  often  demonstrated  on  a prototype  system,  it  is  likely  to  need 
some  development  and  less  likely  than  an  FCT  to  proceed  directly  to  a milestone  C 
decision.  Examples  and  instructions  to  initiate  an  ACTD  can  be  found  at  http:// 
www.acq.osd.mil/actd.  Current  Air  Force  direction  is  under  revision  in  a draft  (AFI  10- 
2302),  Advanced  Concept  Technology  Demonstration. 


The  Air  Forte  established  its  battlelab  program 
to  rapidly  identify  and  demonstrate  the 
military  utility  of  innovative  near-term 
< ontepts  for  the  warfighter. 


The  Air  Force  established  its  battlelab  program  to  rapidly  identify  and  demonstrate 
the  military  utility  of  innovative  near-term  concepts  for  the  warfighter.  The  Air  Force 
stood  the  program  up  with  the  signing  of  Air  Force  Policy  Directive  (AFPD)  10-19, 
Air  Force  Battlelab  Policy,  October  1,  1997.  This  established  the  initial  six  battlelabs: 
1)  Air  Expeditionaiy  Force  (AEF)  Battlelab,  2)  Command  and  Control  Battlelab  (C2B), 
3)  Force  Protection  Battlelab,  4)  Information  Warfare  Battlelab,  5)  Space  Battlelab, 
and  6)  Unmanned  Aerial  Vehicle  Battlelab.  A seventh,  the  Air  Mobility  Battlelab,  was 
added  later.  The  HQ  USAF  Deputy  Chief  of  Staff  (DCS)  for  Warfighting  Integration, 
AF  Battlelabs  Innovation  Division,  provides  overarching  battlelab  guidance,  policy, 
and  oversight.  AFPD  10-19  (1997)established  a close  relationship  between  the 
battlelabs  and  both  Air  Force  Research  Laboratories  (AFRL)  and  Air  Force  Materiel 
Command  (AFMC)  from  the  start.  AFRL  provides  the  battlelabs  with  technical 
expertise;  demonstration  facilities;  data  analysis;  demonstration  ideas;  and  full-time, 
on-site  representation  at  the  battlelabs.  The  battlelabs  offer  AFRL  the  opportunity  to 
match  current  operational  needs  with  existing  research  efforts  in  battlelab  demon- 
strations. AFMC  provides  transitional  support  to  both  the  battlelabs  and  AFRL,  to 
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further  expedite  fielding  of  successful  initiatives.  Battlelab  initiatives  are  funded  through 
both  procurement  and  operations  and  maintenance  (O&M)  funds.  Again,  the  program 
is  unlikely  to  proceed  directly  to  a milestone  C decision  based  on  testing  a prototype 
system.  Links  to  all  battlelabs  may  be  found  on  the  Space  Battlelab  Web  site,  http:/ 
/www.schriever.af.mil/battlelab.  For  additional  information,  reference  AFI  10-2303. 


OTHER  ACQUISITION  TEST  TOOLS 

A major  concern  of  the  Air  Force  research  laboratories  is  transitioning  basic  research 
into  field-able  systems.  As  a technology  progresses  to  execution  under  6.3  research 
funds,  the  laboratory  is  expected  to  demonstrate  its  viability  in  an  operational  significant 
way.  This  demonstration  takes  place  in  the  form  of  critical  experiments  and  advanced 
technology  demonstrations  (ATDs).  To  engender  advocacy  among  the  user  and  acqui- 
sition communities,  laboratory  program  managers  must  encourage  their  participation 
in  the  planning  and  execution  of  these  tests.  Addressing  user  and  acquisition  concerns 
at  this  point  increases  the  likelihood  of  a successful  transition  to  an  acquisition  pro- 
gram. The  identification  of  show  stopper  issues  at  this  stage  will  also  save  the  Air  Force 
valuable  research,  development,  test,  and  evaluation  (RDT&E)  dollars.  The  Air  Force 
Chief  of  Staff  has  also  established  applied  technology  councils  (ATCs)  to  assist  in 
identifying  funding  sources  for  technologies  that  successfully  complete  ATDs. 


Traditional  T&E  tools,  with  the  adaptations  made 
during  the  last  detade,  have  proven  flexible  enough 
to  address  many  of  the  requirements 
of  the  agile  acquisition  environment 


In  addition  to  the  fore  mentioned  tests,  the  Air  Force  and  OSD  have  many  on- 
going tests,  exercises,  and  experiments.  Each  of  these  offers  opportunities  to  identify 
new  needs  and  examine  potential  solutions.  Astute  program  managers  and  test  directors 
may  gain  access  to  resources  and  environments  that  would  otherwise  be  unavailable 
by  participating  in  these  tests.  Exploiting  them  will  often  also  save  both  time  and 
money,  as  well  as  address  test  requirements  that  may  be  otherwise  untestable.  The 
following  are  a few  of  the  major  efforts  and  their  primary  purpose: 

■ Joint  Test  and  Evaluation  (JT&E)-numerous  multi-year  tests  examining  inter-service 
integration  issues  in  designated  mission  areas, 

■ Joint  Expeditionary  Force  Exercise  (JEFX)-annual/biennial  exercises  focused  on 
CSAF-directed  integration  issues, 
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■ Joint  Interoperability  Testing  (JIT)-certifies  system  as  compliant  with  all  joint 
interoperability  requirements, 

■ Weapon  Systems  Evaluation  Program  (WSEP)-annual  weapons  delivery 
evaluations, 

■ Tactics  Development  and  Evaluation  (TD&E)-evaluations  of  specifically  tasked 
concept  of  operations  and  development  of  potential  alternatives,  and 

■ Foreign  Materiel  Exploitation  (FME)-testing  of  newly  acquired,  foreign  militaiy 
hardware. 


SUMMARY 

Traditional  T&E  tools,  with  the  adaptations  made  during  the  last  decade,  have  proven 
flexible  enough  to  address  many  of  the  requirements  of  the  agile  acquisition  environment. 
With  the  continued  drive  to  improve  the  acquisition  process,  the  T&E  process  must 
continue  to  evolve  and  address  future  acquisition  innovations.  The  incentive  evolution- 
ary test  tools  bring  more  to  the  table  than  funding — visibility.  The  headquarters-level 
sponsorship  required  to  become  an  OSD-  or  Air  Force-level  test  program  can  cany 
effective  systems  into  acquisition  programs,  providing  the  political  clout  necessary  to 
find  the  required  funding.  Both  the  traditional  and  evolutionary  test  tools  will  continue 
to  be  invaluable  tools  in  delivering  operational  capabilities  to  the  warfighter.  Additional 
insight  into  the  scope  and  use  of  these  tests  may  be  found  by  referring  to  referenced 
documents  and  Web  sites. 
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